Image processing apparatus, image processing method, and image processing program

ABSTRACT

In order to accurately remove an unnecessary periodic noise component from an image, a reconstruction unit generates a reconstructed image without a periodic noise component by fitting to a face region detected in an image by a face detection unit a mathematical model generated according a method of AAM using a plurality of sample images representing human faces without a periodic noise component. The periodic noise component is extracted by a difference between the face region and the reconstructed image, and a frequency of the noise component is determined. The noise component of the determined frequency is then removed from the image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and animage processing method for removing a periodic noise component includedin an input image. The present invention also relates to a program forcausing a computer to execute the image processing method.

2. Description of the Related Art

In order to reproduce a photographic image in ideal image quality, imageprocessing such as gradation conversion processing, density correctionprocessing, and sharpness processing has been carried out on an image.Especially, for an image obtained by reading a photograph with ascanner, periodic unevenness is observed therein due to performance ofthe scanner. In addition, moiré is observed in a part of an imageobtained by reading an image including halftone dots. A periodic noisecomponent such as periodic unevenness and moiré included in an image canbe removed by carrying out frequency processing on the image.

For example, in U.S. Patent Application Publication No. 20010012407, amethod has been proposed for removing a periodic noise component in aradiographic image caused by a grid used at the time of radiography.This method reconstructs the image by carrying out wavelet transform onthe image for nullifying a signal component in a frequency bandincluding a component representing the grid and by carrying out inversewavelet transform thereafter.

However, if periodic noise components included in an image are in randomfrequency bands, judgment cannot be made as to whether the noisecomponents are periodic unevenness or moiré or a frequency componentrepresenting a subject. For this reason, a frequency componentrepresenting a subject may be removed if the method described in U.S.Patent Application Publication No. 20010012407 is applied for removing aperiodic noise component. Therefore, application of the method describedin U.S. Patent Application Publication No. 20010012407 cannot remove aperiodic noise component from an image with accuracy.

SUMMARY OF THE INVENTION

The present invention has been conceived based on consideration of theabove circumstances. An object of the present invention is therefore toremove only an unnecessary periodic noise component from an image withaccuracy.

An image processing apparatus of the present invention comprises:

reconstruction means for obtaining a reconstructed image of apredetermined structure by reconstructing an image representing thestructure after fitting a model representing the structure to thestructure in an input image having a periodic noise component, the modelobtained by carrying out predetermined statistical processing on aplurality of images representing the predetermined structure without aperiodic noise component, and the model representing the structure byone or more statistical characteristic quantities and weightingparameter or parameters for weighting the statistical characteristicquantity or quantities according to an individual characteristic of thestructure;

noise component extraction means for extracting the periodic noisecomponent in the structure in the input image, by calculating adifference value between values of pixels corresponding to each other inthe structure in the reconstructed image and in the input image;

noise frequency determination means for determining a frequency of theperiodic noise component having been extracted; and

noise removal means for generating a noise-free image by removing theperiodic noise component of the determined frequency from the inputimage.

An image processing method of the present invention comprises the stepsof:

obtaining a reconstructed image of a predetermined structure byreconstructing an image representing the structure after fitting a modelrepresenting the structure to the structure in an input image having aperiodic noise component, the model obtained by carrying outpredetermined statistical processing on a plurality of imagesrepresenting the predetermined structure without a periodic noisecomponent, and the model representing the structure by one or morestatistical characteristic quantities and weighting parameter orparameters for weighting the statistical characteristic quantity orquantities according to an individual characteristic of the structure;

extracting the periodic noise component in the structure in the inputimage, by calculating a difference value between values of pixelscorresponding to each other in the structure in the reconstructed imageand in the input image;

determining a frequency of the periodic noise component having beenextracted; and

generating a noise-free image by removing the periodic noise componentof the determined frequency from the input image.

An image processing program of the present invention is a program forcausing a computer to execute the image processing method describedabove (that is, a program for causing a computer to function as themeans described above).

The image processing apparatus, the image processing method, and theimage processing program of the present invention will be describedbelow in detail.

As a method of generating the model representing the predeterminedstructure in the present invention, a method of AAM (Active AppearanceModel) can be used. An AAM is one of approaches in interpretation of thecontent of an image by using a model. For example, in the case where ahuman face is a target of interpretation, a mathematical model of humanface is generated by carrying out principal component analysis on faceshapes in a plurality of images to be learned and on information ofluminance after normalization of the shapes. A face in a new input imageis then represented by principal components in the mathematical modeland corresponding weighting parameters, for face image reconstruction(T. F. Cootes et al., “Active Appearance Models”, Proc. EuropeanConference on Computer Vision, vol. 2, pp. 484-498, Springer, 1998;hereinafter referred to as Reference 1).

As an example of the periodic noise component can be listed moiré causedby reading a halftone dot image with a scanner, unevenness caused byperformance of a scanner, and moiré generated in an image obtained byphotography with a stationary grid.

It is preferable for the predetermined structure to be suitable formodeling. In other words, variations in shape and color of thepredetermined structure in images thereof preferably fall within apredetermined range. Especially, it is preferable for the predeterminedstructure to generate the statistical characteristic quantity orquantities contributing more to the shape and color thereof throughstatistical processing thereon. Furthermore, it is preferable for thepredetermined structure to be a main part of image. More specifically,the predetermined structure can be a human face.

The plurality of images representing the predetermined structure may beimages obtained by actually photographing the predetermined structure.Alternatively, the images may be generated through simulation based onan image of the structure having been photographed.

It is preferable for the predetermined statistical processing to bedimension reduction processing that can represent the predeterminedstructure by the statistical characteristic quantity or quantities offewer dimensions than the number of pixels representing thepredetermined structure. More specifically, the predeterminedstatistical processing may be multivariate analysis such as principalcomponent analysis. In the case where principal component analysis iscarried out as the predetermined statistical processing, the statisticalcharacteristic quantity or quantities refers/refer to a principalcomponent/principal components obtained through the principal componentanalysis.

In the case where the predetermined statistical processing is principalcomponent analysis, principal components of higher orders contributemore to the shape and color than principal components of lower orders.

The (predetermined) structure in the input image may be detectedautomatically or manually. In addition, the present invention mayfurther comprise the step (or means) for detecting the structure in theinput image. Alternatively, the structure may have been detected in theinput image in the present invention.

A plurality of models may be prepared for respective properties of thepredetermined structure in the present invention. In this case, thesteps (or means) may be added to the present invention for obtaining anyone or more of the properties of the structure in the input image andfor selecting one of the models according to the property having beenobtained. The reconstructed image can be obtained by fitting theselected model to the structure in the input image.

The properties refer to gender, age, and race in the case where thepredetermined structure is human face. The property may be informationfor identifying an individual. In this case, the models for therespective properties refer to models for respective individuals.

As a specific method of obtaining the property may be listed imagerecognition processing having been known (such as image recognitionprocessing described in Japanese Unexamined Patent Publication No.11(1999)-175724). Alternatively, the property may be inferred orobtained based on information such as GPS information accompanying theimage.

Fitting the model representing the structure to the structure in theinput image refers to calculation for representing the structure in theinput image by the model. More specifically, in the case where themethod of ARM described above is used, fitting the model refers tofinding values of the weighting parameters for the respective principalcomponents in the mathematical model.

According to the image processing apparatus, the image processingmethod, and the image processing program of the present invention, thereconstructed image is obtained by reconstructing the image representingthe predetermined structure after fitting, to the structure in the inputimage including the periodic noise component, the model representing thestructure with use of the statistical characteristic quantity orquantities obtained through the predetermined statistical processing onthe images without a periodic noise component and the weightingparameter or parameters that weight(s) the statistical characteristicquantity or quantities according to an individual characteristic of thestructure. The periodic noise component not including a frequencycomponent representing the predetermined structure is removed from thestructure in the reconstructed image. The periodic noise component inthe structure in the input image is extracted by calculating thedifference value between the pixel values corresponding to each other inthe predetermined structure in the reconstructed image and the inputimage, and the frequency of the periodic noise component is determined.Therefore, even in the case where the periodic noise component in theinput image spreads randomly over a plurality of frequency bands, theperiodic noise component can be extracted and the frequency of theperiodic noise component can be determined with accuracy. Consequently,the noise-free image deprived of only the unnecessary periodic noisecomponent can be obtained in high quality by removing from the inputimage the periodic noise component of the frequency having beenobtained.

In the case where the predetermined structure is human face, a humanface is often a main part of image. Therefore, the periodic noisecomponent can be removed optimally for the main part.

In the case where the step (or the means) for detecting the structure inthe input image is added, automatic detection of the structure can becarried out. Therefore, the image processing apparatus becomes easier tooperate.

In the case where the plurality of models are prepared for therespective properties of the predetermined structure in the presentinvention while the steps (or the means) are added for obtaining theproperty of the structure in the input image and for selecting one ofthe models in accordance with the property having been obtained, if thereconstructed image is obtained by fitting the selected model to thestructure in the input image, the structure in the input image can befit to the model that is more suitable. Therefore, processing accuracyis improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows hardware configuration of a digital photograph printer asan embodiment of the present invention;

FIG. 2 is a block diagram showing functions and a flow of processing inthe digital photograph printer in the embodiment and a digital camera inanother embodiment of the present invention;

FIGS. 3A and 3B show examples of screens displayed on a display of thedigital photograph printer and the digital camera in the embodiments;

FIG. 4 is a block diagram showing details of periodic noise componentremoval processing in one aspect of the present invention;

FIG. 5 is a flow chart showing a procedure for generating a mathematicalmodel of face image in the present invention;

FIG. 6 shows an example of how feature points are set in a face;

FIG. 7 shows how a face shape changes with change in values of weightingcoefficients for eigenvectors of principal components obtained throughprincipal component analysis on the face shape;

FIG. 8 shows luminance in mean face shapes converted from face shapes insample images;

FIG. 9 shows how pixel values in a face change with change in values ofweighting coefficients for eigenvectors of principal components obtainedby principal component analysis on the pixel values in the face;

FIGS. 10A through 10E show how an image changes in the periodic noisecomponent removal processing;

FIG. 11 is a block diagram showing an advanced aspect of the periodicnoise component removal processing in the present invention; and

FIG. 12 shows the configuration of the digital camera in the embodimentof the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the accompanying drawings.

FIG. 1 shows hardware configuration of a digital photograph printer asan embodiment of the present invention. As shown in FIG. 1, the digitalphotograph printer comprises a film scanner 51, a flat head scanner 52,a media drive 53, a network adapter 54, a display 55, a keyboard 56, amouse 57, a hard disc 58, and a photographic print output machine 59,all of which are connected to an arithmetic and control unit 50.

In cooperation with the CPU, a main storage, and various input/outputinterfaces of the arithmetic and control unit 50, the unit controls aprocessing flow regarding an image, such as input, correction,manipulation, and output thereof, by executing a program installed froma recording medium such as a CD-ROM. In addition, the arithmetic andcontrol unit 50 carries out image processing calculation for imagecorrection and manipulation. Periodic noise component removal processingof the present invention is also carried out by the arithmetic andcontrol unit 50.

The film scanner 51 photoelectrically reads an APS negative film or a135-mm negative film developed by a film developer (not shown) forobtaining digital image data P0 representing a photograph image recordedon the negative film.

The flat head scanner 52 photoelectrically reads a photograph imagerepresented in the form of hard copy such as an L-size print, forobtaining digital image data P0.

The media drive 53 obtains digital image data P0 representing aphotograph image recorded in a recording medium such as a memory card, aCD, and a DVD. The media drive 53 can also write image data P2 to beoutput therein. The memory card stores image data representing an imagephotographed by a digital camera, for example. The CD or the DVD storesdata of an image read by the film scanner regarding a printing orderplaced before.

The network adapter 54 obtains image data P0 from an order receptionmachine (not shown) in a network photograph service system having beenknown. The image data P0 are image data used for a photograph printorder placed by a user, and sent from a personal computer of the uservia the Internet or via a photograph order reception machine installedin a photo laboratory.

The display 55 displays an operation screen for input, correction,manipulation, and output of an image by the digital photograph printer.A menu for selecting the content of operation and an image to beprocessed are displayed thereon, for example. The keyboard 56 and themouse 57 are used for inputting an instruction.

The hard disc 58 stores a program for controlling the digital photographprinter. In the hard disc 58 are also stored temporarily the image dataP0 obtained by the film scanner 51, the flat head scanner 52, the mediadrive 53, and the network adapter 54, in addition to image data P1having been subjected to image correction (hereinafter referred to asthe corrected image data P1) and the image data P2 having been subjectedto image manipulation (the image data to be output).

The photograph print output machine 59 carries out laser scanningexposure of a photographic printing paper, image development thereon,and drying thereof, based on the image data P2 representing the image tobe output. The photograph print output machine 59 also prints printinginformation on the backside of the paper, cuts the paper for each print,and sorts the paper for each order. The manner of printing may be alaser exposure thermal development dye transfer method.

FIG. 2 is a block diagram showing functions of the digital photographprinter and the flow of processing carried out therein. As shown in FIG.2, the digital photograph printer comprises image input means 1, imagecorrection means 2, image manipulation means 3, and image output means 4in terms of the functions. The image input means 1 inputs the image dataP0 of an image to be printed. The image correction means 2 uses theimage data P0 as input, and carries out automatic image qualitycorrection of the image represented by the image data P0 (hereinafter,image data and an image represented by the image data are represented bythe same reference number) through image processing according to apredetermined image processing condition. The image manipulation means 3uses the corrected image data P1 having been subjected to the automaticcorrection as input, and carries out image processing according to aninstruction from an operator. The image output means 4 uses theprocessed image data P2 as input, and outputs a photographic print oroutputs the processed image data P2 in a recording medium.

The image correction means 2 carries out processing such as gradationcorrection, density correction, color correction, sharpness correction,and white balance adjustment, in addition to the periodic noisecomponent removal processing of the present invention. The imagemanipulation means 3 carries out manual correction on a result of theprocessing carried out by the image correction means 2. In addition, theimage manipulation means 3 carries out image manipulation such astrimming, scaling, change to sepia image, change to monochrome image,and composition with an ornamental frame.

Operation of the digital photograph printer and the flow of theprocessing therein will be described next.

The image input means 1 firstly inputs the image data P0. In the casewhere an image recorded on a developed film is printed, the operatorsets the film on the film scanner 51. In the case where image datastored in a recording medium such as a memory card are printed, theoperator sets the recording medium in the media drive 53. A screen forselecting a source of input of the image data is displayed on thedisplay 55, and the operator carries out the selection by using thekeyboard 56 or the mouse 57. In the case where film has been selected asthe source of input, the film scanner 51 photoelectrically reads thefilm set thereon, and carries out digital conversion. The image data P0generated in this manner are then sent to the arithmetic and controlunit 50. In the case where hard copy such as a photographic print hasbeen selected, the flat head scanner 52 photoelectrically reads the hardcopy set thereon, and carries out digital conversion. The image data P0generated in this manner are then sent to the arithmetic and controlunit 50. In the case where recording medium such as a memory card hasbeen selected, the arithmetic and control unit 50 reads the image dataP0 stored in the recording medium such as a memory card set in the mediadrive 53. In the case where an order has been placed in a networkphotograph service system or by a photograph order reception machine ina store, the arithmetic and control unit 50 receives the image data P0via the network adapter 54. The image data P0 obtained in this mannerare temporarily stored in the hard disc 58.

The image correction means 2 then carries out the automatic imagequality correction on the image represented by the image data P0. Morespecifically, publicly known processing such as gradation correction,density correction, color correction, sharpness correction, and whitebalance adjustment is carried out based on a setup condition set on theprinter in advance, according to an image processing program executed bythe arithmetic and control unit 50. The periodic noise component removalprocessing of the present invention is also carried out, and thecorrected image data P1 are output to be stored in a memory of thearithmetic and control unit 50. Alternatively, the corrected image dataP1 may be stored temporarily in the hard disc 58.

The image manipulation means 3 thereafter generates a thumbnail image ofthe corrected image P1, and causes the display 55 to display thethumbnail image. FIG. 3A shows an example of a screen displayed on thedisplay 55. The operator confirms displayed thumbnail images, andselects any one of the thumbnail images that needs manual image-qualitycorrection or order processing for image manipulation while using thekeyboard 56 or the mouse 57. In FIG. 3A, the image in the upper leftcorner (DSCF0001) is selected. As shown in FIG. 3B as an example, theselected thumbnail image is enlarged and displayed on the display 55,and buttons are displayed for selecting the content of manual correctionand manipulation on the image. The operator selects a desired one of thebuttons by using the keyboard 56 or the mouse 57, and carries outdetailed setting of the selected content if necessary. The imagemanipulation means 3 carries out the image processing according to theselected content, and outputs the processed image data P2. The imagedata P2 are stored in the memory of the arithmetic and control unit 50or stored temporarily in the hard disc 58. The program executed by thearithmetic and control unit 50 controls image display on the display 55,reception of input from the keyboard 56 or the mouse 57, and imageprocessing such as manual correction and manipulation carried out by theimage manipulation means 3.

The image output means 4 finally outputs the image P2. The arithmeticand control unit 50 causes the display 55 to display a screen for imagedestination selection, and the operator selects a desired one ofdestinations by using the keyboard 56 or the mouse 57. The arithmeticand control unit 50 sends the image data P2 to the selected destination.In the case where a photographic print is generated, the image data P2are sent to the photographic print output machine 59 by which the imagedata P2 are output as a photographic print. In the case where the imagedata P2 are recorded in a recording medium such as a CD, the image dataP2 are written in the CD or the like set in the media drive 53.

The periodic noise component removal processing of the present inventioncarried out by the image correction means 2 will be described below indetail. FIG. 4 is a block diagram showing details of the periodic noisecomponent removal processing. As shown in FIG. 4, the periodic noisecomponent removal processing is carried out by a face detection unit 31,a reconstruction unit 32, a noise component extraction unit 33, a noisefrequency determination unit 34, and a noise removal unit 35. The facedetection unit 31 detects a face region P0 f in the image P0. Thereconstruction unit 32 fits to the detected face region P0 f amathematical model M generated by a method of AAM (see Reference 1above) based on a plurality of sample images representing human faceswithout a periodic noise component, and obtains a reconstructed image P1f by reconstructing the face region having been subjected to thefitting. The noise component extraction unit 33 extracts a periodicnoise component N0 in the face region P0 f by calculating a differencein values of pixels corresponding to each other in the face region P0 fand in the reconstructed image P1 f. The noise frequency determinationunit 34 determines a frequency of the noise component N0. The noiseremoval unit 35 obtains a noise-free image P1′ by removing the noisecomponent of the determined frequency from the image P0. The image P1′is an image subjected only to the noise removal processing, and theimage P1 is the image having been subjected to all the processing suchas the gradation correction and the white balance adjustment. Theprocessing described above is controlled by the program installed in thearithmetic and control unit 50.

The mathematical model M is generated according to a flow chart shown inFIG. 5, and installed in advance together with the programs describedabove. Hereinafter, how the mathematical model M is generated will bedescribed.

For each of the sample images representing human faces without aperiodic noise component, feature points are set as shown in FIG. 6 forrepresenting face shape (Step #1). In this case, the number of thefeature points is 122. However, only 60 points are shown in FIG. 6 forsimplification. Which part of face is represented by which of thefeature points is predetermined, such as the left corner of the left eyerepresented by the first feature point and the center between theeyebrows represented by the 38^(th) feature point. Each of the featurepoints may be set manually or automatically according to recognitionprocessing. Alternatively, the feature points may be set automaticallyand later corrected manually upon necessity.

Based on the feature points set in each of the sample images, mean faceshape is calculated (Step #2). More specifically, mean values ofcoordinates of the feature points representing the same part are foundamong the sample images.

Principal component analysis is then carried out based on thecoordinates of the mean face shape and the feature points representingthe face shape in each of the sample images (Step #3). As a result, anyface shape can be approximated by Equation (1) below:

$\begin{matrix}{S = {S_{0} + {\sum\limits_{i = 1}^{n}{p_{i}b_{i}}}}} & (1)\end{matrix}$

S and S0 are shape vectors represented respectively by simply listingthe coordinates of the feature points (x1, y1, . . . , x122, y122) inthe face shape and in the mean face shape, while pi and bi are aneigenvector representing the i^(th) principal component for the faceshape obtained by the principal component analysis and a weightcoefficient therefor, respectively. FIG. 7 shows how face shape changeswith change in values of the weight coefficients b1 and b2 for theeigenvectors p1 and p2 as the highest and second-highest order principalcomponents obtained by the principal component analysis. The changeranges from −3 sd to +3 sd where sd refers to standard deviation of eachof the weighting coefficients b1 and b2 in the case where the face shapein each of the sample images is represented by Equation (1). The faceshape in the middle of 3 faces for each of the components represents theface shape in the case where the values of the weighting coefficientsare the mean values. In this example, a component contributing to faceoutline has been extracted as the first principal component through theprincipal component analysis. By changing the weighting coefficient b1,the face shape changes from an elongated shape (corresponding to −3 sd)to a round shape (corresponding to +3 sd). Likewise, a componentcontributing to how much the mouth is open and to length of chin hasbeen extracted as the second principal component. By changing the weightcoefficient b2, the face changes from a state of open mouth and longchin (corresponding to −3 sd) to a state of closed mouth and short chin(corresponding to +3 sd). The smaller the value of i, the better thecomponent explains the shape. In other words, the i^(th) componentcontributes more to the face shape as the value of i becomes smaller.

Each of the sample images is then subjected to conversion (warping) intothe mean face shape obtained at Step #2 (Step #4). More specifically,shift values are found between each of the sample images and the meanface shape, for the respective feature points. In order to warp pixelsin each of the sample images to the mean face shape, shift values to themean face shape are calculated for the respective pixels in each of thesample images according to 2-dimensional 5-degree polynomials (2) to (5)using the shift values having been found:

$\begin{matrix}{x^{\prime} = {x + {\Delta\; x}}} & (2) \\{y^{\prime} = {y + {\Delta\; y}}} & (3) \\{{\Delta\; x} = {\sum\limits_{i = 0}^{n}{\sum\limits_{j = 0}^{n - i}{a_{ij} \cdot x^{i} \cdot y^{j}}}}} & (4) \\{{\Delta\; y} = {\sum\limits_{i = 0}^{n}{\sum\limits_{j = 0}^{n - i}{b_{ij} \cdot x^{i} \cdot y^{j}}}}} & (5)\end{matrix}$

In Equations (2) to (5) above, x and y denote the coordinates of each ofthe feature points in each of the sample images while x′ and y′ arecoordinates in the mean face shape to which x and y are warped. Theshift values to the mean shape are represented by Δx and Δy with n beingthe number of dimensions while aij and bij are coefficients. Thecoefficients for polynomial approximation can be found by using a leastsquare method. At this time, for a pixel to be moved to a positionrepresented by non-integer values (that is, values including decimals),pixel values therefor are found through linear approximation using 4surrounding points. More specifically, for 4 pixels surroundingcoordinates of the non-integer values generated by warping, the pixelvalues for each of the 4 pixels are determined in proportion to adistance thereto from the coordinates generated by warping. FIG. 8 showshow the face shape of each of 3 sample images is changed to the meanface shape.

Thereafter, principal component analysis is carried out, using asvariables the values of RGB colors of each of the pixels in each of thesample images after the change to the mean face shape (Step #5). As aresult, the pixel values of RGB colors in the mean face shape convertedfrom any arbitrary face image can be approximated by Equation (6) below:

$\begin{matrix}{A = {A_{0} + {\sum\limits_{i = 1}^{m}{q_{i}\lambda_{i}}}}} & (6)\end{matrix}$

In Equation (6), A denotes a vector (r1, g1, b1, r2, g2, b2, . . . , rm,gm, bm) represented by listing the pixel values of RGB colors at each ofthe pixels in the mean face shape (where r, g, and b represent the pixelvalues of RGB colors while 1 to m refer to subscripts for identifyingthe respective pixels with m being the total number of pixels in themean face shape). The vector components are not necessarily listed inthis order in the example described above. For example, the order may be(r1, r2, . . . , rm, g1, g2, . . . , gm, b1, b2, . . . , bm). A0 is amean vector represented by listing mean values of the RGB values at eachof the pixels in the mean face shape while qi and λi refer to aneigenvector representing the i^(th) principal component for the RGBpixel values in the face obtained by the principal component analysisand a weight coefficient therefor, respectively. The smaller the valueof i is, the better the component explains the RGB pixel values. Inother words, the component contributes more to the RGB pixel values asthe value of i becomes smaller.

FIG. 9 shows how faces change with change in values of the weightcoefficients λi1 and λi2 for the eigenvectors qi1 and qi2 representingthe i1^(th) and i2^(th) principal components obtained through theprincipal component analysis. The change in the weight coefficientsranges from −3 sd to +3 sd where sd refers to standard deviation of eachof the values of the weight coefficients λi1 and λi2 in the case wherethe pixel values in each of the sample face images are represented byEquation (6) above. For each of the principal components, the face inthe middle of the 3 images corresponds to the case where the weightcoefficients λi1 and λi2 are the mean values. In the examples shown inFIG. 9, a component contributing to presence or absence of beard hasbeen extracted as the i1^(th) principal component through the principalcomponent analysis. By changing the weight coefficient λi1, the facechanges from the face with dense beard (corresponding to −3 sd) to theface with no beard (corresponding to +3 sd). Likewise, a componentcontributing to how a shadow appears on the face has been extracted asthe i2^(th) principal component through the principal componentanalysis. By changing the weight coefficient λi2, the face changes fromthe face with a shadow on the right side (corresponding to −3 sd) to theface with a shadow on the left side (corresponding to +3 sd). How eachof the principal components contributes to what factor is determinedthrough interpretation.

In this embodiment, the plurality of face images representing humanfaces have been used as the sample images. Therefore, in the case wherea component contributing to difference in face luminance has beenextracted as the first principal component, luminance in the face regionP0 f in the image P0 is changed with change in the value of theweighting coefficient λ1 for the eigenvector q1 of the first principalcomponent, for example. The component contributing to the difference inface luminance is not necessarily extracted as the first principalcomponent. In the case where the component contributing to thedifference in face luminance has been extracted as the K^(th) principalcomponent (K≠1), “the first principal component” in the descriptionbelow can be replaced by “the K^(th) principal component”. Thedifference in luminance in face is not necessarily represented by asingle principal component. The difference may be due to a plurality ofprincipal components.

Through the processing from Step #1 to #5 described above, themathematical model M can be generated. In other words, the mathematicalmodel M is represented by the eigenvectors pi representing the faceshape and the eigenvectors qi representing the pixel values in the meanface shape, and the number of the eigenvectors is far smaller for pi andfor qi than the number of pixels forming the face image. In other words,the mathematical model M has been compressed in terms of dimension. Inthe example described in Reference 1, 122 feature points are set for aface image of approximately 10,000 pixels, and a mathematical model offace image represented by 23 eigenvectors for face shape and 114eigenvectors for face pixel values has been generated through theprocessing described above. By changing the weight coefficients for therespective eigenvectors, approximately 98% of variations in face shapeand pixel values can be expressed.

A flow of the periodic noise component removal processing based on theAAM method using the mathematical model M will be described next, withreference to FIG. 4 and FIGS. 10A to 10E. FIGS. 10A to 10E show how animage changes in the periodic noise component removal processing.

The face detection unit 31 reads the image data P0, and detects the faceregion P0 f in the image P0. More specifically, the face region can bedetected through various known methods such as a method using acorrelation score between an eigen-face representation and an image ashas been described in Published Japanese Translation of a PCTApplication No. 2004-527863 (hereinafter referred to as Reference 2).Alternatively, the face region can be detected by using a knowledgebase, characteristics extraction, skin-color detection, templatematching, graph matching, and a statistical method (such as a methodusing neural network, SVM, and HMM), for example. Furthermore, the faceregion P0 f may be specified manually with use of the keyboard 56 andthe mouse 57 when the image P0 is displayed on the display 55.Alternatively, a result of automatic detection of the face region may becorrected manually. FIG. 10A shows the image P0 while FIG. 10B shows theface region P0 f. The image P0 and the face region P0 f have a periodicnoise component shown by diagonal lines.

The reconstruction unit 32 fits the mathematical model M to the faceregion P0 f for reconstructing the face region P0 f. More specifically,the image is reconstructed according to Equations (1) and (6) describedabove while sequentially changing the values of the weight coefficientsbi and λi for the eigenvectors pi and qi corresponding to the principalcomponents in order of higher order in Equations (1) and (6). The valuesof the weighting coefficients bi and λi causing a difference between thereconstructed image and the face region P1 f to become minimal are thenfound (see Reference 2 for details). It is preferable for the values ofthe weighting coefficients bi and λi to range only from −3 sd to +3 sdwhere sd refers to the standard deviation in each of distributions of biand λi when the sample images used at the time of generation of themodel are represented by Equations (1) and (6). In the case where thevalues do not fall within the range, it is preferable for the weightingcoefficients to take the mean values in the distributions. In thismanner, erroneous application of the model can be avoided.

The reconstruction unit 32 obtains the reconstructed image P1 f by usingthe weighting coefficients bi and λi having been found. FIG. 10C showsthe reconstructed image P1 f. As shown in FIG. 10C, the reconstructedimage P1 f does not include the periodic noise component, since themathematical model M has been generated from the sample images without aperiodic noise component.

The noise component extraction unit 33 then finds as the noise componentN0 a difference value between values of pixels corresponding to eachother in the face region P0 f and in the reconstructed image P1 f. Morespecifically, a value of a pixel in the reconstructed image P1 f issubtracted from a value of the corresponding pixel in the face region P0f, for finding the difference value. FIG. 10D shows the periodic noisecomponent N0. As shown in FIG. 10D, the periodic noise component N0represents a noise component in the region corresponding to the faceregion P0 f.

The noise frequency determination unit 34 then determines the frequencyof the periodic noise component N0. More specifically, the periodicnoise component is decomposed into a plurality of frequency componentsthrough frequency transform processing such as Fourier transform andwavelet transform carried out on the periodic noise component N0. Afrequency band in which the noise component exists is the frequency ofthe noise. The frequency of the periodic noise component N0 maycorrespond only to one frequency band or a plurality of frequency bands.Therefore, a plurality of frequencies may be determined as the frequencyof the noise in some cases.

The noise removal unit 35 removes from the image P0 the noise componentof the frequency having been determined, for generating the noise-freeimage P1′. More specifically, the periodic noise component N0 isdecomposed into a plurality of frequency components through frequencytransform processing such as Fourier transform and wavelet transformcarried out on the image P0, and processing is carried out for reducing,preferably nullifying, the frequency component in the frequency bandcorresponding to the frequency having been determined. The frequencycomponent after the processing is subjected to inverse frequencytransform, in order to generate the noise-free image P1′. FIG. 10E showsthe noise-free image P1′. As shown in FIG. 10E, the periodic noisecomponent included in the image P0 has been removed in the image P1′.

As has been described above, according to the periodic noise componentremoval processing in the embodiment of the present invention, thereconstruction unit 32 reconstructs the face region P0 f by fitting, tothe face region P0 f detected by the face detection unit 31 in the imageP0, the mathematical model M generated according to the method of ARMbased on the sample images representing human faces not including aperiodic noise component. The reconstruction unit 32 then generates thereconstructed image P1 f from which the periodic noise component notincluding a frequency component of the face region P0 f has beenremoved. By calculating the difference value between the correspondingpixel values in the reconstructed image P1 f and in the face region P0f, the periodic noise component N0 in the face region P0 f is extracted,and the frequency of the periodic noise component N0 is determined.Therefore, even in the case where the periodic noise component in theimage P0 exists randomly over a plurality of frequency bands, only theperiodic noise component that is unnecessary can be extracted withaccuracy, for determining the frequency of the periodic noise componentwith precision. Consequently, by removing the noise component of thedetermined frequency from the image P0, the image P1′ from which theperiodic noise component not including the frequency component of theface region P0 f has been removed with accuracy can be obtained in highquality. As a result, the image P2 can be obtained in high quality.

In the embodiment described above, the mathematical model M is unique.However, a plurality of mathematical models Mi (i=1, 2, . . . ) may begenerated for respective properties such as race, age, and gender, forexample. FIG. 11 is a block diagram showing details of periodic noisecomponent removal processing in this case. As shown in FIG. 11, aproperty acquisition unit 36 and a model selection unit 37 are added,which is different from the embodiment shown in FIG. 4. The propertyacquisition unit 36 obtains property information AK of a subject in theimage P0. The model selection unit 37 selects a mathematical model MKgenerated only from sample images representing subjects having aproperty represented by the property information AK.

The mathematical models Mi have been generated based on the same method(see FIG. 5), only from sample images representing subjects of the samerace, age, and gender, for example. The mathematical models Mi arestored by being related to property information Ai representing each ofthe properties that is common among the samples used for the modelgeneration.

The property acquisition unit 36 may obtain the property information AKby judging the property of the subject through execution of knownrecognition processing (such as processing described in JapaneseUnexamined Patent Publication No. 11(1999)-175724) on the image P0.Alternatively, the property of the subject may be recorded at the timeof photography as accompanying information of the image P0 in a headeror the like so that the recorded information can be obtained. Theproperty of the subject may be inferred from accompanying information.In the case where GPS information representing a photography location isavailable, the country or a region corresponding to the GPS informationcan be identified. Therefore, the race of the subject can be inferred tosome degree. By paying attention to this fact, a reference tablerelating GPS information to information on race may be generated inadvance. By inputting the image P0 obtained by a digital camera thatobtains the GPS information at the time of photography and records theGPS information in a header of the image P0 (such as a digital cameradescribed in Japanese Unexamined Patent Publication No. 2004-153428),the GPS information recorded in the header of the image data P0 isobtained. The information on race related to the GPS information may beinferred as the race of the subject when the reference table is referredto according to the GPS information.

The model selection unit 37 obtains the mathematical model MK related tothe property information AK obtained by the property acquisition unit36, and the reconstruction unit 32 fits the mathematical model MK to theface region P0 f in the image P0.

As has been described above, in the case where the mathematical modelsMi corresponding to the properties have been prepared, if the modelselection unit 37 selects the mathematical model MK related to theproperty information AK obtained by the property acquisition unit 36 andif the reconstruction unit 32 fits the selected mathematical model MK tothe face region P0 f, the mathematical model MK does not haveeigenvectors contributing to variations in face shape and luminancecaused by difference in the property information AK. Therefore, the faceregion P0 f can be represented only by eigenvectors representing factorsdetermining the face shape and luminance other than the factorrepresenting the property. Consequently, processing accuracy improves.

From a viewpoint of improvement in processing accuracy, it is preferablefor the mathematical models for respective properties to be specifiedfurther so that a mathematical model for each individual as a subjectcan be generated. In this case, the image P0 needs to be related toinformation identifying each individual.

In the embodiment described above, the mathematical models are installedin the digital photograph printer in advance. However, from a viewpointof processing accuracy improvement, it is preferable for mathematicalmodels for different human races to be prepared so that which of themathematical models is to be installed can be changed according to acountry or a region to which the digital photograph printer is going tobe shipped.

The function for generating the mathematical model may be installed inthe digital photograph printer. More specifically, a program for causingthe arithmetic and control unit 50 to execute the processing describedby the flow chart in FIG. 5 is installed therein. In addition, a defaultmathematical model may be installed at the time of shipment thereof. Themathematical model may be customized based on input images to thedigital photograph printer, or a new model different from the defaultmodel may be generated. This is especially effective in the case wherethe models for respective individuals are generated.

In the embodiment described above, the individual face image isrepresented by the face shape and the weighting coefficients bi and λifor the pixel values of RGB colors. However, the face shape iscorrelated to variation in the pixel values of RGB colors. Therefore, anew appearance parameter c can be obtained for controlling both the faceshape and the pixel values of RGB colors as shown by Equations (7) and(8) below, through further execution of principal component analysis ona vector (b1, b2, . . . , bi, . . . , λ1, λ2, . . . , λi, . . . )combining the weighting coefficients bi and λ i:S=S ₀ +Q _(S) c  (7)A=A ₀ +Q _(A) c  (8)

A difference from the mean face shape can be represented by theappearance parameter c and a vector QS, and a difference from the meanpixel values can be represented by the appearance parameter c and avector QA.

In the case where this model is used, the reconstruction unit 32 findsthe face pixel values in the mean face shape based on Equation (8) abovewhile changing a value of the appearance parameter c. Thereafter, theface image is reconstructed by conversion from the mean face shapeaccording to Equation (7) above, and the value of the appearanceparameter c causing a difference between the reconstructed face imageand the face region P0 f to be minimal is found.

As another embodiment of the present invention can be installation ofthe periodic noise component removal processing in a digital camera. Inother words, the periodic noise component removal processing isinstalled as an image processing function of the digital camera. FIG. 12shows the configuration of such a digital camera. As shown in FIG. 12,the digital camera has an imaging unit 71, an A/D conversion unit 72, animage processing unit 73, a compression/decompression unit 74, a flashunit 75, an operation unit 76, a media recording unit 77, a display unit78, a control unit 70, and an internal memory 79. The imaging unit 71comprises a lens, an iris, a shutter, a CCD, and the like, andphotographs a subject. The A/D conversion unit 72 obtains digital imagedata P0 by digitizing an analog signal represented by charges stored inthe CCD of the imaging unit 71. The image processing unit 73 carries outvarious kinds of image processing on image data such as the image dataP0. The compression/decompression unit 74 carries out compressionprocessing on image data to be stored in a memory card, and carries outdecompression processing on image data read from a memory card in acompressed form. The flash unit 75 comprises a flash and the like, andcarries out flash emission. The operation unit 76 comprises variouskinds of operation buttons, and is used for setting a photographycondition, an image processing condition, and the like. The mediarecording unit 77 is used as an interface with a memory card in whichimage data are stored. The display unit 78 comprises a liquid crystaldisplay (hereinafter referred to as the LCD) and the like, and is usedfor displaying a through image, a photographed image, various settingmenus, and the like. The control unit 70 controls processing carried outby each of the units. The internal memory 79 stores a control program,image data, and the like.

The functions of the image input means 1 in FIG. 2 are realized by theimaging unit 71 and the A/D conversion unit 72. Likewise, the functionsof the image correction means 2 are realized by the image processingunit 73 while the functions of the image manipulation means 3 arerealized by the image processing unit 73, the operation unit 76, and thedisplay unit 78. The functions of the image output means 4 are realizedby the media recording unit 77. All of the functions described above arerealized under control by the control unit 70 with use of the internalmemory 79.

Operation of the digital camera and a flow of processing therein will bedescribed next.

The imaging unit 71 causes light entering the lens from a subject toform an image on a photoelectric surface of the CCD when a photographerfully presses a shutter button. After photoelectric conversion, theimaging unit 71 outputs an analog image signal, and the A/D conversionunit 72 converts the analog image signal output from the imaging unit 71to a digital image signal. The A/D conversion unit 72 then outputs thedigital image signal as the digital image data P0. In this manner, theimaging unit and the A/D conversion unit 72 function as the image inputmeans 1.

Thereafter, the image processing unit 73 carries out gradationcorrection processing, density correction processing, color correctionprocessing, white balance adjustment processing, and sharpnessprocessing in addition to the periodic noise component removalprocessing, and outputs corrected image data P1. In this manner, theimage processing unit 73 functions as the image correction means 2. Inorder to realize the periodic noise component removal processing, thecontrol unit 70 starts a periodic noise component removal program storedin the internal memory 79, and causes the image processing unit 73 tocarry out the periodic noise component removal processing (see FIG. 4)using the mathematical model M stored in advance in the internal memory79, as has been described above.

The image P1 is displayed on the LCD by the display unit 78. As a mannerof this display can be used display of thumbnail images as shown in FIG.3A. While operating the operation buttons of the operation unit 76, thephotographer selects and enlarges one of the images to be processed, andcarries out selection from a menu for further manual image correction ormanipulation. Processed image data P2 are then output. In this manner,the functions of the image manipulation means 3 are realized.

The compression/decompression unit 74 carries out compression processingon the image data P2 according to a compression format such as JPEG, andthe compressed image data are written via the media recording unit 77 ina memory card inserted in the digital camera. In this manner, thefunctions of the image output means 4 are realized.

By installing the periodic noise component removal processing of thepresent invention as the image processing function of the digitalcamera, the same effect as in the case of the digital photograph printercan be obtained.

The manual correction and manipulation may be carried out on the imagehaving been stored in the memory card. More specifically, thecompression/decompression unit 74 decompresses the image data stored inthe memory card, and the image after the decompression is displayed onthe LCD of the display unit 78. The photographer selects desired imageprocessing as has been described above, and the image processing unit 73carries out the selected image processing.

Furthermore, the mathematical models for respective properties ofsubjects described by FIG. 11 may be installed in the digital camera. Inaddition, the processing for generating the mathematical model describedby FIG. 5 may be installed therein. A person as a subject of photographyis often fixed to some degree for each digital camera. Therefore, if amathematical model is generated for the face of each individual as afrequent subject of photography with the digital camera, a model withoutvariation of individual difference in face can be generated.Consequently, the periodic noise component removal processing can becarried out with extremely high accuracy for the face of the person.

The program of the present invention may be incorporated with imageediting software for causing a computer to execute the periodic noisecomponent removal processing. In this manner, a user can use theperiodic noise component removal processing of the present invention asan option of image editing and manipulation on his/her computer, byinstallation of the software from a recording medium such as a CD-ROMstoring the software to the personal computer, or by installation of thesoftware through downloading of the software from a predetermined Website on the Internet.

1. An image processing apparatus comprising: reconstruction means forobtaining a reconstructed image of a predetermined structure byreconstructing an image representing the structure after fitting a modelrepresenting the structure to the structure in an input image having aperiodic noise component, the model obtained by carrying outpredetermined statistical processing on a plurality of imagesrepresenting the predetermined structure without a periodic noisecomponent, and the model representing the structure by one or morestatistical characteristic quantities and weighting parameter orparameters for weighting the statistical characteristic quantity orquantities by correlating the shape and the variation in the pixel valueof the predetermined structure according to an individual characteristicof the structure; noise component extraction means for extracting theperiodic noise component in the structure in the input image bycalculating a difference value between values of pixels corresponding toeach other in the structure in the reconstructed image and in the inputimage; noise frequency determination means for determining a frequencyof the periodic noise component having been extracted; noise removalmeans for generating a noise-free image by removing the periodic noisecomponent of the determined frequency from the input image; and furthercomprising selection means for obtaining a property of the structure inthe input image and for selecting the model corresponding to theobtained property from a plurality of the models representing thestructure for respective properties of the predetermined structure,wherein the reconstruction means obtains the reconstructed image byfitting the selected model to the structure in the input image, whereinthe selection means obtains information about a location of photography,and the selected model comprising a best match further takes intoaccount race based on the location of photography.
 2. An imageprocessing method using a processor comprising: obtaining areconstructed image of a predetermined structure by reconstructing animage representing the structure after fitting models representing thestructure to the structure in an input image having a periodic noisecomponent, the models obtained by carrying out predetermined statisticalprocessing on a plurality of images representing the predeterminedstructure without a periodic noise component, and the model representingthe structure by one or more statistical characteristic quantities andweighting parameter or parameters for weighting the statisticalcharacteristic quantity or quantities by correlating the shape and thevariation in the pixel value of the predetermined structure according toan individual characteristic of the structure; extracting the periodicnoise component in the structure in the input image by calculating adifference value between values of pixels corresponding to each other inthe structure in the reconstructed image and in the input image;determining a frequency of the periodic noise component having beenextracted; generating a noise-free image by removing the periodic noisecomponent of the determined frequency from the input image; obtaining aproperty of the structure in the input image; selecting the modelcorresponding to the obtained property from a plurality of the modelsrepresenting the structure for respective properties of thepredetermined structure, and wherein reconstructing the reconstructedimage comprises fitting the selected model to the structure in the inputimage, further including obtaining information about a location ofphotography, and selecting the model comprises taking into account racebased on the location of photography.
 3. An image processing programembodied in a non-transitory computer readable medium for causing acomputer to function as: reconstruction means for obtaining areconstructed image of a predetermined structure by reconstructing animage representing the structure after fitting a model representing thestructure to the structure in an input image having a periodic noisecomponent, the model obtained by carrying out predetermined statisticalprocessing on a plurality of images representing the predeterminedstructure without a periodic noise component, and the model representingthe structure by one or more statistical characteristic quantities andweighting parameter or parameters for weighting the statisticalcharacteristic quantity or quantities by correlating the shape and thevariation in the pixel value of the predetermined structure according toan individual characteristic of the structure; noise componentextraction means for extracting the periodic noise component in thestructure in the input image by calculating a difference value betweenvalues of pixels corresponding to each other in the structure in thereconstructed image and in the input image; noise frequencydetermination means for determining a frequency of the periodic noisecomponent having been extracted; noise removal means for generating anoise-free image by removing the periodic noise component of thedetermined frequency from the input image; and selection means forobtaining a property of the structure in the input image and forselecting the model corresponding to the obtained property from aplurality of the models representing the structure for respectiveproperties of the predetermined structure, wherein the reconstructionmeans obtains the reconstructed image by fitting the selected model tothe structure in the input image, wherein the selection means obtainsinformation about a location of photography, and the selected modelcomprising a best match takes into account race based on the location ofphotography.